Skip to main content
Feedback

Setting up Google Cloud Storage as a target

Set your Google Cloud credential JSON key, create a Google Cloud Storage bucket, and get the necessary credentials for using Google Cloud Storage with Data Integration.

Prerequisite

Ensure you’ve signed up for the Google Platform and you have a console Admin user. If you do not have one of these prerequisites, you can start here .

Create a service account user for Data Integration

Data Integration uses a Google Cloud Storage bucket to upload your source data into it. Create a user in the Google Cloud Platform console with access to the relevant bucket and the relevant BigQuery project.

Procedure

  1. Sign in to the Google Cloud Platform console.
  2. Go to IAM & Admin  > Service account > and click CREATE SERVICE ACCOUNT.
  3. In the CREATE SERVICE ACCOUNT page:
    1. Set your Service Account name (Data_Integration User) and click CREATE AND CONTINUE.
    2. Grant the service account access to the project by setting Roles:
      • Click on the drop-down list and select BigQuery Admin.
      • Click the ADD ANOTHER ROLE and do the same process for Storage Admin.
      • Copy your Account Service ID / Email from the service account list. You can use this to enter it in a Data Integration connection.
  4. Create a key for the service account:
    1. Go to the service account screen, locate the service you created, and click on it.
    2. In the new service account page, click on Key.
    3. Click on Add key.
    4. Choose key type JSON and click on create.
    5. Your JSON secret key download is complete.

Enabling Cloud Storage and GCS API

  1. Go to API's & Services and click ENABLE APIS AND SERVICE.
  2. Search for Google Cloud Storage JSON API and click Enable API.

Creating a Google Cloud Storage bucket

Data Integration needs a Google Cloud Storage bucket as a FileZone before loading your data to BigQuery. You can use the FileZone bucket or objects as a base for other Hadoop or Apache Spark operations, such as those provided by Google Data PROC or your different services.

Procedure

  1. Sign in to the Google Cloud Platform console.
  2. Go to Storage > Browse, and click CREATE BUCKET.
  3. In the CREATE BUCKET page: 
    1. Set Bucket Name, for example: project_name_data_integration_file_zone.
    2. Set your Bucket to Regional (Multi-Region is not stable for loading) and choose your preferred location.
    3. Click CREATE.

Configuring Google Cloud Storage bucket in Data Integration

Create a new connection for your Google Cloud Storage. Enter your credentials information for Google Platform Service Account.

  1. Connection Name
  2. Project ID  (is available on the Google Platform Home section)
  3. Project Number (optional - is available on Google Platform Home section)
  4. Service Account Email  - you need the Service Account ID to copy the Service Account page.
  5. Region: indicates the bucket’s creation location.
  6. Set your custom File Zone to save the data in your own staging area (Optional).
  7. Click Test Connection. After validating it, save the connection.

Known issues

Sometimes the Storage Admin type user role does not have a certain permission storage.buckets.get given to it by default:

In this case, you must edit your GCP user roles by duplicating that Storage Admin role. Ensure the custom role you create has the storage.buckets.get permission, then assign your service account this custom role instead of the Storage Admin (Refer to the Create a Service Account User for Data Integration section).

On this Page